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Abstract 

Let N be a connected and simply connected nilpotent Lie group, A 
a lattice in N, and A\iV the corresponding nilmanifold. Let AfT (A\N) 
be the group of affine transformations of A\N. 

We characterize the countable subgroups H of Aff ( A\ A~) for which 
the action of H on A\N has a spectral gap, that is, such that the 
associated unitary representation U° of H on the space of functions 
from L 2 (A\N) with zero mean does not weakly contain the trivial 
representation. Denote by T the maximal torus factor associated to 
A\N . We show that the action of H on A\N has a spectral gap if and 
only if there exists no proper ii-invariant subtorus S of T such that 
the projection of H on Aut (T/S) has an abelian subgroup of finite 
index. 

We first establish the result in the case where A\N is a torus. In 
the case of a general nilmanifold, we study the asymptotic behaviour 
of matrix coefficients of U° using decay properties of metaplectic rep- 
resentations of symplectic groups. The result shows that the existence 
of a spectral gap for subgroups of AS(A\N) is equivalent to strong 
ergodicity in the sense of K. Schmidt. Moreover, we show that the 
action of H on A\AT is ergodic (or strongly mixing) if and only if the 
corresponding action of H on T is ergodic (or strongly mixing). 

1 Introduction 

Let if be a countable group acting measurably on a probability space (X, v) 
by measure preserving transformations. Let U : h i— >■ U(h) denote the corre- 
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sponding Koopman representation of if on L 2 (X, v). We say that the action 
of if on X has a spectral gap if the restriction U° of U to the if-invariant 
subspace 

L 2 (X,v) = {teL 2 (X,v) : / £{x)dv{x) = 0} 

Jx 

does not have almost invariant vectors, that is, there is no sequence of unit 
vectors £ n in f q(A, v) such that lim n \\U°(h)C, n — £ n || =0 for all h £ if. A 
useful equivalent condition for the existence of a spectral gap is as follows. 
Let /i be a probability measure on if such that the support of \x generates 
H . Let U°(fjj) be the convolution operator defined on f q(A, v) by 

fee// 

Observe that we have ||£7°(/i)|| < 1 and hence r(U°(fi)) < 1 for the spectral 
radius r(U°(fj)) of U°(fj). Assume that /x is aperiodic, (that is, if supp(/i) is 
not contained in the coset of a proper subgroup of if). Then the action of 
if on A has a spectral gap if and only if r(U°([i)) < 1 and this is equivalent 
to \\U°(n)\\ < 1. 

Ergodic theoretic applications of the existence of a spectral gap (or of the 
stable spectral gap; see below for the definition) to random walks (such as 
the rate of L 2 -convergence in the random ergodic theorem, pointwise ergodic 
theorem, analogues of the law of large numbers and of the central limit theo- 
rem, etc) are given in [UoGullj . [UoLellj . |FuSh99j . [GoNelO] and |Guiv05j . 
Another application of the spectral gap property is the uniqueness of v as 
if -invariant mean on L°°(X,u); for this as well as for further applications, 
see |BeHV08j . |Lubo94j . |Popa08| , |Sarn90j . 

Recall that a factor (Y, m, H) of the system (A, v, H) is a probability 
space (Y, m) equipped with an if- act ion by measure preserving transforma- 
tions together with a if -equivariant mesurable mapping $ : A — > Y with 
$*(z/) = m. Observe that L 2 (Y,m) can be identified with a if -invariant 
closed subspace of L 2 (A, v). 

By a result proved in |JuRo79[ Theorem 2.4], no action of a countable 
amenable group by measure preserving transformations on a non-atomic 
probability space has a spectral gap. As a consequence, if there exists a 
non-atomic factor (Y, m, if) of the system (A, v, if) such that if acts as an 
amenable group on Y, then the action of if on A has no spectral gap. Our 
main result (Theorem [1]) shows in particular that this is the only obstruction 
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for the existence of a spectral gap when H is a countable group of affine 
transformations of a compact nilmanifold X. 

Let N be a connected and simply connected nilpotent Lie group. Let A 
be a lattice in N; the associated nilmanifold A\N is known to be compact. 
The group A" acts by right translations on A\A" : every n E N defines a 
transformation p(n) on A\A" given by As i-> Axn. Denote by Aut(AQ the 
group of continuous automorphisms of A" and by Aut(A\AQ the subgroup of 
continuous automorphisms if of N such that (p(A) = A. The group Aut(A^) is 
a linear algebraic group defined over Q and Aut(A\AT) is a discrete subgroup 
of Aut(AT). An affine transformation of A\A" is a mapping A\A" — > A\N of 
the form ip o p(n) for some <p G Aut(A\AT) and n G N. The group Aff (A\AT) 
of affine transformations of A\A" is the semi-direct product Aut(A\AT) x N. 

Every g G AS(A\N) preserves the translation invariant probability mea- 
sure z/a\tv induced by a Haar measure on N. The action of AS(A\N) on 
A\A" is a natural generalization of the action of SL n (Z) x T n on the torus 
T n = Rny Z n_ In factj let j> = A[N, N]\N be the maximal torus factor of 
A\A". Then the nilsystem (A\N, H) can be viewed as the result, starting with 
T, of a finite sequence of extensions by tori, with induced actions of H on 
every stage. 

Actions of of higher rank lattices by affine transformations on nilmani- 
folds arise in Zimmer's programme as one of the standard actions for such 
groups (see the survey [Fish] ) . The action of a single affine transforma- 
tion (or a flow of such transformations) on a nilmanifold have been stud- 
ied by W. Parry from the ergodic, spectral or topological point of view (see 
[Parr69j . |Parr70-a] . |Parr70-b] ; see also |AuGH63] for the case of translations) . 

Let V be a finite dimensional real vector space and A a lattice in V. As is 
well-known, T = V/ A is a torus and A defines a rational structure on V. Let 
W be a rational linear subspace of V. Then S = W/(W(~) A) is a subtorus of 
T and we have a torus factor T = T/S. Let H be a subgroup of Aff(T) and 
assume that W is invariant under p^H), where p a : Aff (A\AT) — > Aut(A\AT) 
is the canonical projection. Then H leaves S invariant and the induced 
action of H on T is a factor of the action of H on T. We will say that T is 
an if- invariant factor torus of T. Here is our main result. 

Theorem 1 Let A\N be a compact nilmanifold with associated maximal 
torus factor T. Let H be a countable subgroup Aff(A\A r ). The following 
properties are equivalent: 

(i) The action of H on A\N has a spectral gap. 



3 



(ii) The action of H on T has a spectral gap. 

(Hi) There exists no non-trivial H -invariant factor torus T of T such that 
the projection of p a (H) on Aut(T) is a virtually abelian group (that is, 
it contains an abelian subgroup of finite index). 

To give an an example, let T = R d /Z d be the d-dimensional torus. Ob- 
serve that Aut(T) can be identified with GL^(Z). Let H be a subgroup 
of Aff(T) = GL^(Z) ix T. Assume that Pa,(H) is not virtually abelian and 
that p a (H) acts Q-irreducibly on R d (that is, there is no non-trivial p a (H)- 
invariant rational subspace of R d ). Then the action of H on T has a spectral 
gap. For more details, see Corollary |6] and Example [7] below. 

The result above is new even in the case where A\A is a torus; see 
however |FuSh99t Theorem 6.5.ii] for a sufficient condition for the existence 
of a spectral gap for groups of torus automorphisms. Our results shows, in 
particular, that the spectral gap property for a countable subgroup H of 
Aff(A\iV) is equivalent to the spectral gap property for its automorphism 
part p a (H). 

The proof of Theorem [1] breaks into two parts. We first establish the 
result in the case where A\N is a torus (see Theorem [5] below ). Our proof 
is based here on the existence of appropriate invariant means on finite di- 
mensional vector spaces. A crucial tool will be (a version of) Furstenberg's 
result on stabilizers of probability measures on projective spaces over local 
fields. In the case of a general nilmanifold A\N with associated maximal 
torus factor T, we show that (ii) implies (i) by studying the asymptotic 
behaviour of matrix coefficients of the Koopman representation U of H re- 
stricted to the orthogonal complement of L 2 (T) in L 2 (A\N); for this, we will 
use decay properties of the metaplectic representation of symplectic groups 
due to R. Howe and C. C.Moore [H oMo79] . The equivalence of (i) and (ii) 
was proved in [BcHclOj in the special case of a group of automorphisms of 
Heisenberg nilmanifolds. 

Actions of countable amenable groups on a non-atomic probability space 
fail to have a property which is weaker than the spectral gap property. Re- 
call that the action of a countable group H by measure preserving trans- 
formations on a probability space (X, v) is said to be strongly ergodic in 
Schmidt's sense (see [Schm80] . [Schm81] ) if every sequence (A n ) n of measur- 
able subsets of X which is asymptotically invariant (that is, which is such that 
lim„ u(gA n AA n ) = for all g G H) is trivial (that is, lim„ v(A n )(l— u(A n )) = 
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0). It is easy to see that if the action of if on X has a spectral gap, then the 
action is strongly ergodic (see, for instance, |BeHV08| Proposition 6.3.2]). 
The converse does not hold in general (see Example (2.7) in |Schm8lj ). As 
shown in [Schm8lj . no action of a countable amenable group by measure pre- 
serving transformations on a non-atomic, probability space can be strongly 
ergodic. 

An interesting feature of strong ergodicity (as opposed to the spectral 
gap property) is that this notion only depends on the equivalence relation 
on X defined by the partition of X into if-orbits. Our result shows that the 
existence of a spectral gap for subgroups of Aff (A\iV) is equivalent to strong 
ergodicity. 

Corollary 2 The action of a countable subgroup of Aff(A\N) on a compact 
nilmanifold A\N has a spectral gap if and only if it is strongly ergodic. 

We suspect that the previous corollary is true for every countable group 
of affine transformations of the quotient of a Lie group by a lattice. In fact, 
the following stronger statement could be true. Let G be a connected Lie 
group and T a lattice of G. Let if be a countable subgroup of Aff(r\G). 
Assume that the action of H on r\G does not have a spectral gap. Is it true 
that there exists a non-trivial if -invariant factor T\G of r\G such that the 
closure of the projection of H on Aff(r\G) is an amenable group? 

As our result shows, this is indeed the case if G is a nilpotent Lie group; it 
is also the case if G is a simple non-compact Lie group with finite centre (see 
Theorem 6.10 in [FuSh 99j). It is worth mentioning that the corresponding 
statement in the framework of countable standard equivalence relations has 
been proved in [JoSc87j. 

Let again if be a countable group acting by measure preserving transfor- 
mations on a probability space (X, v). The following useful strengthening of 
the spectral gap property has been considered by several authors ( |Bekk90] . 
[BeGu06j . |FuSh99| . |Popa08| ). Following |Popa08| , let us say that the action 
of if has a stable spectral gap if the diagonal action of if on (X xl,i/®i/) has 
a spectral gap (see Lemma 3.2 in |Popa08| for the rationale of this terminol- 
ogy). The following result is an immediate consequence of Theorem [1] above 
and of the corresponding result for groups of torus automorphisms obtained 
in [FuSh99l Theorem 6.4]. 

Corollary 3 If the action of a countable subgroup of Aff(A\A) on a compact 
nilmanifold A\N has a spectral gap, then it is has stable spectral gap. 
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Next, we turn to the question of the ergodicity or mixing of the action of 
a (not necessarily countable) subgroup H of Aff(A\iV) on A\N . As a con- 
sequence of our methods, we will see that this reduces to the same question 
for the action of H on the associated torus. 

Recall that an action of a group H on a probability space (X, v) is weakly 
mixing if the Koopman representation U of H on L 2 (X, v) has no finite 
dimensional subrepresentation, and that the action of of a countable group 
H is strongly mixing if the matrix coefficients g h- >■ (U(g)£,r)) vanish at 
infinity for all £, 77 G Lq(X, v). 

Theorem 4 Let H be a group of affine transformations of the compact nil- 
manilfold A\N. Let T be the maximal T torus factor associated to A\N. 

(i) If the action of H on T is ergodic (or weakly mixing), then its action 
on A\N is ergodic (or weakly mixing). 

(ii) Assume that H is as subgroup of Aut(A\iV). If the action of H on T 
is strongly mixing, then its action on A\N is strongly mixing. 

Part (i) of the previous theorem has been independently established in 
|CoGull] ) with a different method of proof. In the case of a single affine 
transformation (that is, in the case of H = Z), the result is due to W. Parry 
(see |Parr69j . |Parr70-a] ). Also, |CoGull] gives an example of a group of 
automorphisms H acting ergodically on a nilmanilfold A\N for which no 
single automorphism from H acts ergodically on A\N, showing that the 
previous theorem does not follow from Parry's result. 

Sections 1-7 are devoted to the proof our main result Theorem [T] in the 
case where A\N is a torus. The proof of the extension to general nilmanifold 
is given in Sections 8-14. Theorem H] is treated in Section 15. 

Acknowlegments We are grateful to J-P. Conze, A. Furman, and A. Gam- 
burd for useful discussions. 

2 Spectral gap property for groups of affine 
transformations of a torus: statement of 
the main result 

Let V be a finite dimensional real vector space of dimension d > 1 and 
let A be a lattice in V. Let T be the torus T — V/A. The group of affine 
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transformations of T is the semi-direct product Aff(T) = Aut(T) ix T. 

The aim of this section is to state the following result, which will be proved 
in the next two sections. Recall that p a denotes the canonical homomorphism 
Aff(T) Aut(T). 

Theorem 5 Let H be a countable subgroup of AS (T). The following prop- 
erties are equivalent. The following properties are equivalent: 

(i) The action of H on T does not have a spectral gap. 

(ii) There exists a non-trivial H -invariant factor torus T such that the 
projection of Pa.(H) on Aut(T) is amenable. 

(Hi) There exists a non-trivial H -invariant factor torus T Q such that the 
projection ofp a (H) on Aut(T ) is virtually abelian. 

The following corollary is an immediate consequence of the implication (z) =>- 
(Hi) in the previous theorem. 

Corollary 6 Let T = V/A be a torus. Let H be a countable subgroup of 
Aff(T) such that p a (H) C Aut(T) is not virtually abelian. Assume that the 
action of H on V is Q-irreducible for the rational structure on V defined by 
A. Then the action of H on T has a spectral gap. 

This last result was proved in |FuSh99l Theorem 6.5.ii] for a subgroup H 
of Aut(T) under the stronger assumption that the action of H on V is R- 
irreducible. We give an example of a subgroup H of automorphisms of a 6- 
dimensional torus T = V/A which acts Q-irreducibly but not R-irreducibly 
on V and which has a spectral gap on T. 

Example 7 Let q be the quadratic form on R 3 given by 

q(x) = x\ + x\ — V2x1, 

and let SO(q, R) C GL 3 (R) be the orthogonal group of q. Set 

H = SL 3 (Z[V2] n SO(q,TL). 

Let a be the non-trivial automorphism of the field Q[\/2]. For every g E 
SO(q, R), the matrix g a , obtained by conjugating each entry of g, preserves 
the conjugate form q a of q under a. The mapping 

Q[\/2]4RxR, x^(x,a(x)) 
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induces an isomorphism between Z[-\/2] 3 and a lattice A in R 3 xR 3 . It induces 
also an isomophism 7 \-t (7, 7°") between H and a lattice T in SO(q, R) x 
SO^, R). Moreover, if leaves Z[-\/2] 3 invariant and T leaves A invariant. 
We obtain in this way an action of H on the torus T = R 6 /A. 

Since SO(q a ,~R) = 50(3) is compact, if is a lattice in SO(q, R). This 
implies (Borel density theorem) that the Zariski closure of H in SX 3 (R) is 
the simple Lie group SO(q, R), so that the action of H on R 3 is R-irreducible 
and hence Q-irreducible for the usual rational structure on R 3 . It follows that 
the action of H on R 6 is Q-irreducible for the rational structure defined by 
the lattice A of R 6 . Observe that the action of H on R 6 is not R-irreducible 
since V leaves invariant each copy of R 3 in R 6 = R 3 ©R 3 . Moreover, H is not 
virtually abelian as it is a lattice in SO(q, R) = S0(2, 1). As a consequence 
of the previous corollary, the action of H on T has a spectral gap. 

Concerning the proof of Theorem |5j we will first treat the case of groups 
of toral automorphisms. 

Choosing a basis for the Z-module A, we identify V with R d and A 
with Z d . By means of the standard scalar product on R d , we identify the 
dual group V of V (that is, the group of unitary characters of V) with 
V. The dual action of an element g G GL(V) on V corresponds to the 
action of (g -1 )* on V. Since T — V/A, the dual group T can be identified 
with A. Let W be a rational linear subspace of V. The dual group of the 
quotient V/W corresponds to the orthogonal complement W 1 - of W, which 
is also a rational linear subspace of V. The dual group of the torus factor 
T = {V/W)/{{W + A) /A) corresponds to W 1 n A. 

The discussion above shows that Theorem [5j in the case of a group of 
toral automorphisms is equivalent to the following theorem. 

Theorem 8 Let H be a subgroup of GLd(7i). The following properties are 
equivalent. 

(i) The action of H on T = R d /Z d does not have a spectral gap. 

(ii) There exists a non-trivial rational subspace W ofH d which is invariant 
under the subgroup H l of GLd(Z) and such that the image of H l in 
GL(W) is an amenable group. 

(Hi) There exists a non-trivial rational subspace W ofH d which is invariant 
under H l and such that the the image of H t in GL(W) is a virtually 
abelian group. 
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Observe that the implication (m) ==>- (ii) is obvious and that the impli- 
cation (ii) ==>• (i) follows from the result in [JuRo79j quoted in the introduc- 
tion. Therefore, it remains to show that (i) implies (ii) and that (ii) implies 
(hi). 

3 A canonical amenable group associated to 
a linear group 

Let V be a finite-dimensional real vector space. (Although we will consider 
only real vector spaces, the results in this section are valid for vector spaces 
over any local field.) Let g G GL(V) and W a ^-invariant linear subspace of V. 
We denote by gw £ GL(W) the automorphism of W given by the restriction 
of g to W. If W is another ^-invariant subspace contained in W, we will 
denote by gw/w £ GL(W/W) the automorphism of W/W induced by g. 
Also, if if is a subgroup of GL(V) and W C W are ii-invariant subspaces of 
V, we will denote by H w and H w / W i the corresponding subgroups of GL(W) 
and GL(W/W), respectively. 

For a subgroup H of GL(V), we denote by H its closure for the usual 
locally compact topology on GL(V). The aim of this section is to prove the 
following result. 

Proposition 9 Let H be a subgroup of GL(V). There exists a largest H- 
invariant linear subspace V(H) of V such that the group Hy(H) is amenable. 
More precisely, let V(H) be the subspace of V generated by the union of the 
H -invariant subspaces W C V for which Hw is amenable. Then Hy(H) is 
amenable. 

A more explicit description of V(H) will be given later (Proposition [T5l) . 
For the proof of the proposition above, we will need the following elementary 
lemma. 

Lemma 10 Let H be a closed subgroup of GL(V) and W an H -invariant 
subspace ofV. Then H is amenable if and only if Hw and Hy/w ore amenable. 

Proof Since Hw and Hy/w are closures of quotients of H, both are 
amenable if H is amenable. 
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Assume that H w and H v /w are amenable. Let L be the closed subgroup 
consisting of the elements g G GL(V) leaving W invariant and for which gw 
belongs to H w and gv/w belongs to H v /w- The mapping 

(p : L H w x H V / W , g (g w , gv/w) 

is a continuous homomorphism. It is clear that (p is surjective. Moreover, 
U = Ker(cp) is a unipotent closed subgroup of L. Since Hw x H v /w an d 
[/ are amenable, L is amenable. The closed subgroup H of L is therefore 
amenable. ■ 

Proof of Proposition O We can write V(H) = YH=i as a sum °f finitely 
many if- invariant subspaces Wi, . . . , W r of V such that Hy/ l is amenable for 
every 1 < i < r. 

We show by induction on s G {1, . . . ,r} that is amenable, where 
W s = J2i=i Wi- The case s — 1 being obvious, assume that i?vy s is amenable 
for some s G {1, . . . , r — 1}. The group 

GL(W S+1 /W S ) = GL((W S + W s+1 )/W s ) 

is canonically isomorphic to GL(W s+ i/ (W s PI W^+i) and Hw*+i/w» corre- 
sponds to -ffvF s+ i/(H/ a nVK s+ i) under this isomorphism. Now, H Ws+1 /(y/s nWa+1 ^ is 
amenable since Hw 3+1 is amenable. Hence, H W s+i/ W s is amenable. Moreover, 
Hw* is amenable by the induction hypothesis. The previous lemma implies 
that H W s+\ is amenable. ■ 

4 Invariant means supported by rational sub- 
spaces 

Let G be a locally compact group. There is a well-known relationship between 
weak containment properties of the trivial representation 1q and existence 
on invariant means on appropriate spaces (see below). We will need to make 
this relationship more precise in the case where if is a subgroup of toral 
automorphisms. 

By a unitary representation {tt,T-L) of G, we will always mean a strongly 
continuous homomorphism 7r : G — > U (H) from G to the unitary group of a 
complex Hilbert space H. 



10 



Recall that, for every finite measure p of G, the operator tt(//) G £>(7f) is 
defined by the integral 



J G 

Assume that G is a discrete group and 7r and p are unitary representations of 
G; then 7r is weakly contained in p if and only if ||7r(/z)|| < for every 

finite measure p on G (see Section 18 in |Dixm69j ) . Recall also that, given a 
probability measure p on G which is aperiodic, the trivial representation 1^ 
is weakly contained in a unitary representation 7r if and only if ||7r(/i)|| = 1 
(see |BeHV08l G.4.2]). 

Let X be a topological space and C b (X) the Banach space of all bounded 
continuous functions on X equipped with the supremum norm. Recall that a 
mean on X is a linear functional m on C b (X) such that m(lx) = 1 and such 
that m(ip) > for every <p G C b (X) with <p > 0. A mean is automatically 
continuous. We will often write m(A) instead of m(l^) for a subset A of X. 

Observe that the means on a compact space X are the probability mea- 
sures on X. 

Let if be a group acting on X by homeomorphisms. Then if acts nat- 
urally on C b (X). A mean m on X is if -invariant if m(h.(p) = m(ip) for all 



Let Y be another topological space and / :I->Ka continuous mapping. 
For every mean m on X, the push-forward f*(m) of m is the mean on Y 
defined by <p \-> m((p o f) for <p G C b (Y). 

We will consider invariant means on two kinds of topological spaces: 

• X is a set with the discrete topology and endowed with an action of a group 
H. It is well-known (see Theoreme on p. 44 in |Eyma72] ) that there exists 
an if-invariant mean on X if and only if the natural unitary representation 
U of if on £ 2 (X) almost has invariant vectors (that is, if and only if U weakly 
contains the trivial representation 1# of H). 

• X = V \ {0}, where V is a finite dimensional real vector space. Let H be 
a subgroup of GL(V). If m is an if-invariant mean on V \ {0}, then 7r*(m) 
is an if -invariant probability measure on the projective space P(V), where 
7r : V \ {0} — > P(V) is the canonical projection. 

The following result is a version of Furstenberg's celebrated lemma (see 
[Furs76] or |Zimm84| Corollary 3.2.2]) on stabilizers of probability measures 
on projective spaces. We will need later (in Section [5]) the more precise form 
we give for this lemma (see also the proof of Theorem 6.5 (ii) in [FuSh99j). 




for all £eU. 



ip E C b 



(X) and /i G if. 
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For a subgroup H of GL(V), we denote by Zc(H) the closure of H in the 
Zariski topology and by Zc(H)° the connected component of Zc(H) in the 
Zariski topology. As is well-known, Zc(H)° has finite index in Zc(H). 

Lemma 11 Let H be a closed subgroup of GL(V). Assume that H stabilizes 
a probability measure v on P{V) which is not supported on a proper projec- 
tive subspace. Then the commutator subgroup [H°, H°] of H° is relatively 
compact, where H° is the normal subgroup of finite index H n Zc(H)° of H. 
In particular, H is amenable. 

Proof We can find finitely many positive measures (z^)i<j< r on P(V) with 
v = J2i<i< r v i sucn that v (Vi ^ Vj) = for i 7^ j and such that supp(z/j) C 
7r(Vi) for every i 6 {l,...,r}, where Vi is a linear subspace of V of minimal 
dimension with Vi(7r(Vi)) > 0. The if-orbit of Vi and hence the if-orbit 
of Vi is finite (see Proof of Corollary 3.2.2 in |Zimm84] ) . Since stabilizers of 
probability measures on P(V) are algebraic (see Theorem 3.2.4 in [Zimm84] ) . 
it follows that H° stabilizes each Vi and each z/j. Now z/j, viewed as measure 
on P(Vj), is zero on every proper projective subspace of P(Vi). Hence (see 
Corollary 3.2.2 in |Zimm84] ). the image of the restriction H® of H° to Vi 
is a relatively compact subgroup of PGL(Vi), for every i e {1, . . . ,r}. Since 



[H?, Hf] is contained in SL(Vj), it follows that [iff, gg] is compact in GL(^). 



This implies that [H°,H°] is compact. As H°/[H°,H°] is abelian, it follows 
that H° (and hence H) is amenable. ■ 

Remark 12 The conclusion of the previous lemma does not hold in general 
if we replace H° by an arbitrary subgroup of finite index of H. For example, 
let V = Rei © Re2 and let H C GZ^R) be the stabilizer of the measure 
v — (^(ei) + ^7r(e 2 ))/2 on P(V). Then [H, H] = H is not bounded; however, 
if is the subgroup of index two consisting of the diagonal matrices in H and 
[H°, H°] is trivial. 

Proposition 13 Let H be a subgroup of GL(V) and V(H) the largest H- 
invariant susbpace of V such that Hy(H) is amenable. 

(i) Assume H stabilizes a mean m on V \ {0}. Then V(H) ^ {0}. 

(ii) Let A be a lattice in V and m a mean on A \ {0}. Assume H leaves A 
invariant and stabilizes m. Then m(V(H) D A) = 1. In particular, the 
H-linear span ofV(H) R A is a non-trivial rational subspace ofV (for 
the rational structure defined by A). 
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Proof (i) Let 7r : V \ {0} — > P(V) be the canonical projection and v = 
71"* (m). Then v is an if -invariant probability measure on P{V). Let W the 
linear span of 7r _1 (supp(z/)). Then W is non-trivial and v is not supported on 
a proper projective subspace of 7r(W). It follows from Lemma [TT1 applied to 
the closed subgroup Hw of GL(W) that Hw is amenable. Hence, V(H) ^ 
{0}, by the definition of V(H). 

(ii) Set V = V/V(H). Since V(H) is if-invariant, we have an induced 
action of H on V. Denote by p : V — > V the canonical projection. We 
consider the mean m = (p|a)*(^) on the set A := p(A) equipped with the 
discrete topology. Observe that m is if -invariant, since H stabilizes m. 

Assume, by contradiction, that m(V(H) n A) < 1. Then m({0}) = 
m(V(H) fl A) < 1. Setting a = m(V(H) D A), we define an if-invariant 
mean ral on A \ {0} by 

mi(m) = -J—m( v ) for all (p G £°°(A \ {0}). 
1 — a 

Let (mi) be the mean on V \ {0} induced by the canonical injection i : 
A \ {0} — > V \ {0}. Observe that i*(m"i) is ii-invariant. Hence, by (i), we 
have V(H) ^ {0}. This implies that V(H) is a proper subspace of the vector 
space W := p~ 1 (V(ii)). On the other hand, H\y is amenable, by Lemma [TOl 
This contradicts the definition of V(H). ■ 

At this point, we can give the proof of the fact that (i) implies (ii) in 
Theorem [5] (or, equivalently, in Theorem [8]) in the case of group of automor- 
phisms. 

Proof of (i) =^ (ii) in Theorem [5] 

Let if be a countable subgroup of GLd(Z). Assume that the action of H 
on T = R d /Z d does not have a spectral gap. Then the unitary representation 
of the transposed subgroup H l on £ 2 (Z d \ {0}) weakly contains the trivial 
representation l#t. Hence, there exists an if*-invariant mean on Z d \ {0}. 
By Proposition [13], the linear span W of V(H l ) fl Z d is a non-trivial rational 
subspace of R d . Morever, = is amenable. ■ 



5 Proof of (ii) =^> (in) in Theorem [8 

For the proof of (ii) (Hi) in Theorem E] we will need a precise description 
of the subspace V(H) associated to a subgroup H of GL(V) and introduced 
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in Proposition HI For this, we will use the following result which appears as 
Lemma 1 and Lemma 2 in [CoGu74] . Since the arguments in |CoGu74] are 
slightly incomplete, we give the proof of this lemma. 

Lemma 14 Let V be finite- dimensional real vector space and let H be a 
subgroup of GL(V) such that the action of H on V is completely reducible. 

(i) Assume that the eigenvalues of every element in H all have modulus 1. 
Then H is relatively compact. 

(ii) Assume that there exists an integer N > 1 such that the eigenvalues of 
every element in H are all N-th roots of unity. Then H is finite. 

Proof By hypothesis, we can decompose V into a direct sum V = ®i<i< r Vi 
of irreducible if-invariant subspaces V{. Let V c = V ®r C be the complex- 
ification of V. The action of H on each Vi extends to a representation of 
H on Vp which either is irreducible or decomposes as a direct sum of two 
irreducible (mutually conjugate) representations of H. It suffices therefore to 
prove the following 

Claim: Let H be a subgroup of GL d (C) acting irreducibly on C d . Then the 
conclusion (i) and (ii) hold. 

For every h G H, we consider the linear functional tph on the algebra 
M d (C) of complex (d x <i)-matrices defined by iph{x) = Tr(hx). Since H 
acts irreducibly, it follows from Burnside theorem that the algebra generated 
by H coincides with M d {C). Hence, there exists a basis {hi, . . . , h d 2} of the 
vector space M d {C) contained in H. Then {(phn ■ ■ ■ yVh d2 } is a basis of the 
dual space of M d {C). 

Assume that the eigenvalues of every element in H all have modulus 1. 
Then the (ph. 's are bounded on H by d. It follows that the matrix coefficients 
of the elements in H are bounded. Hence, H is relatively compact subset of 
M d (C). 

Assume that, for a fixed N > 1, the eigenvalues of every element in H 
are N-th. roots of unity. Then the take only a finite set of values on H. 
It follows that H is finite subset of M d (C)M 

Proposition 15 Let V be a finite- dimensional real vector space and H a sub- 
group H of GL(V). Set H° = Hr\Zc(H)°. Let V 1 be the largest H -invariant 
linear subspace ofV such that, for every h e [H°,H°], the eigenvalues of the 
restriction of h to V 1 all have modulus 1. Then V(H) = V 1 . 
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Proof Let us first show that V(H) C V 1 . Since Hytm is amenable, there 
exists an if- invariant probability measure v on P(V(H)) C P(V). Let W 
be the smallest if- invariant subspace such that v is supported on P(W). It 
follows from Lemma [TT1 that [if , if ] acts isometrically on W, with respect to 
an appropriate norm on W. We can apply the same argument to the group 
Hv{H)/w acting on the quotient space V(H)/W. Hence, by induction, we 
obtain a flag 

{0} = Wo C W = W l C W 2 C • • ■ C W r = V(H) 

of if-invariant subspaces such that [if , if ] acts isometrically on each quo- 
tient Wi+i/Wi. It follows from this that the eigenvalues of the restriction to 
V(H) of any element h G [if , if ] have all modulus 1. Hence, V(H) C V 1 . 

To show that V 1 C V(H), we have to prove that Hyi is amenable. Recall 
that that if/if is finite and observe that H vl /[H vl , ff°i] is abelian. Hence, 

it suffices to show that [if^ifyx] is amenable. 
Let 

{0} = W C Wx C ■ • • C W r = V 1 

be a Jordan- Holder sequence for the [ifyi, ifyi]-module V 1 , that is, ev- 
ery Wi is an [Hy X , ify X ]-invariant subspace of V 1 and [ifyi,if°i] acts ir- 
reducibly on every quotient Wi+i/Wi. By Lemma UMi. the image of [if , if ] 
in GL(Wi+i/Wi) is relatively compact for every i e {0, . . . , r — 1}. 

Let N be the unipotent subgroup of GL (V 1 ) consisting of the elements 
in GL (V 1 ) which act trivially on every quotient W i+ i/Wi. 

We can choose a scalar product on V 1 such that, denoting by W^ the 
orthogonal complement of Wi in W^+i, every h G [if , if ] can be written in 
the form h = kho, where ho G iV and where k leaves W^ invariant and acts 
isometrically on W^ for every i G {0, . . . , r — 1}, This shows that [ff°i, ff°i] 
can be embedded as a closed subgroup of K x iV C GL (V 1 ), where K is the 
product of the the orthogonal groups of the W^s. Since K tx N is amenable, 
the same is true for [ff°i, ff°i]- • 

We will need need the following corollary of (the proof of) the previous 
proposition . 

Corollary 16 Let Y be a subgroup of GL^Z). Assume that the eigenvalues 
of every 7 G T all have modulus 1. Then V contains a unique maximal 
unipotent subgroup T of finite index. In particular, T° is a characteristic 
subgroup ofY. 
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Proof As in the proof of the previous proposition, we consider a Jordan- 
Holder sequence for the r-module R d 

{0} = Wo C Wi C ■ • • C W r = R d 

and let N be the subgroup of all g G GL(V) which act trivially on every 
Wi+i/Wi. We choose a scalar product on R d such that T embeds as a sub- 
group of the semi-direct product K x iV for K = Y[t=i 0(W^), where is 
the orthogonal complement of Wi in Wj+i. 

Let 7 G T. For every I > 1, the /-th powers of the eigenvalues of 7 are 
roots of the same monic polynomial with integer coefficients and of degree 
d. Since the eigenvalues of 7 are all of modulus 1, the coefficients of this 
polynomial are bounded by a number only depending on d. By a standard 
argument (see e.g. the proof of Lemma 11.6 in |StTa87j ). it follows that all 
the eigenvalues of 7 are roots of unity of a fixed order iV which only depends 
on d. 

Let T be the projection of T in K. The action of V is completely reducible, 
since the W^s are irreducible, and it follows from Lemma [THii that T is 
finite. Hence, r R N is a unipotent normal subgroup of finite index in F. 

We have therefore proved that T contains a unipotent normal subgroup of 
finite index. We claim that T := rnZc(T) is the unique maximal unipotent 
normal subgroup of finite index in T. 

Indeed, let Ti be a unipotent normal subgroup of finite index in T. Set 
U := Zc(Ti). Observe that the connected component of U coincides with 
Zc(r)°, since r\ has finite index in T. On the other hand, as is well-known, 
U is connected since it is a unipotent algebraic group. (Indeed, the Zariski 
closure of the subgroup generated by a unipotent element u G GL(R, d ) con- 
tains the one-parameter subgroup through u; see e.g. 15.1. Lemma C in 
[Hum81] .) It follows that Zc(r)° — U is unipotent. Moreover, since T\ C U, 
we have Ti C T and the claim is proved. ■ 

We can now complete the proof of Theorem [HI 
Proof of (ii) =^ (iii) in Theorem [8] 

Let T = V/A be a torus and H a subgroup of Aut(T) C GL(V). Assume 
that there exists a non-trivial rational subspace W of V which is iY-invariant 
and such that such that the restriction Hw of H to W is an amenable group. 
In particular, we have W C V(H). 

Set H° = H n Zc(H)°. By Proposition M> for every h G [H°, H°], all the 
eigenvalues of the restriction of h to W have modulus 1. Since W is rational, 
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by the choice of a convenient basis of W, we can assume that T := [H°, H°]w 
is a subgroup of GL^(Z), where d = dimly. It follows from Corollary [TBI 
that T contains a unipotent subgroup T° of finite index which is moreover 
characteristic. Let W\ be the space of the r°-fixed vectors in W. Then W\ is 
a rational and non-trivial linear subspace of W. Moreover, W\ is if-invariant, 
since T is characteristic. 

We claim that Hy/ 1 is virtually abelian. For this, it suffices to show that 
G := Hyy, C GL{W\) is virtually abelian. Observe first that [G,G] = Yy/ 1 is 
finite, since it is a quotient of the finite group T/T°. Since [Zc(G), Zc(G)] C 
Zc([G, G]), it follows that [Zc(G), Zc(G)] is finite. On the other hand, the 
group [Zc(G)°, Zc(G)°] is connected (see e.g. Proposition 17.2 in [Hum81j ). 
Hence, Zc(G)° is abelian. The subgroup G D Zc(G)° has finite index in G 
and is abelian. ■ 

6 Herz's majoration principle for induced rep- 
resentations 

Unitary representations of a separable locally compact group G induced by 
a closed subgroup H will appear several times in the sequel. We review 
their definition when the homogeneous space H\G has G-invariant measure. 
This will always be the case in the situations we will encounter. (Induced 
representation are still defined in the general case, after appropriate change; 
see |Mack76j or |BeHV08j .) 

Let v be non-zero G-invariant measure on H\G. Let (a, K.) be a unitary 
representation of H. We will use the following model for the induced represen- 
tation Ind^cr. Choose a measurable section s : H\G — > G for the canonical 
projection G — > H\G. Let c : (H\G) x G — > H be the corresponding cocycle 
defined by 

s(x)g = c(x, g)s(xg) for all x G H\G, g G G. 

The Hilbert space of Ind^a is the space L 2 (H\G, K.) of all square-integrable 
measurable mappings £ : H\G — > /C and the action of G on L 2 (H\G,JC) is 
given by 

(1nd%<r)(g)£(x) = a{c{x,g))^{xg), geG, £ G L 2 (H\G,1C), x G G/R 
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In the sequel, we will use several times a well-known strengthening of 
Herz's majoration principle from [H erz70] concerning norms of convolution 
operators under an induced representation. For an even more general version, 
see |Anan03[ 2.3.1]. For the convenience of the reader, we give the short proof. 



Proposition 17 (Herz's majoration principle) Let H be a closed sub- 
group of G such that H\G has a G-invariant Borel measure v and let (ex, /C) 
be a unitary representation of H. For every probability measure fi on the 
Borel subsets of G, we have 

\\(lnd G H a)(»)\\ < \\p G/ Ml 

where Xg/h is the natural representation of G on L 2 (G/H). 

Proof Let c : H\G H be the cocycle defined by a Borel section of 
H\G -»■ G. For f e L 2 (H\G, JC,u), define ip in the Hilbert space L 2 (H\G, u), 
of Ind^ex by <p(x) = and observe that \\<p\\ = ||£||. Using Jensen's 

inequality, we have 

jG_\/..\ t ||2 . / ll/T„jG 



\\(lnd%a)(^\\ 2 = / ||(Ind£(/i)£(x))||^) 

JH\G 



H\G JG 



o-(c(x, g))£(xg)dfi(g)\\ 2 dv(x) 



< / \\a{c{x,g))^{xg)\\ 2 dn{g)du(x) 

Jh\g Jg 

M(xg)\\ 2 dfi(g)du(x) 

h\g Jg 
= ||(Ind|lH)(/i)^H 2 . 

Since Ind^l# is equivalent to Xg/h, the claim follows. ■ 

We will also need (in Section [TU]) a precise description of the kernel of an 
induced representation. 

Lemma 18 With the notation as in the previous proposition, let it = Ind^cr. 
Then Ker(7r) = f] g€G gKer(<7)g _1 , that is, Ker(7r) coincides the largest nor- 
mal subgroup of G contained in Kercr. 
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Proof 

Let c : H\G x G — > if be the cocycle corresponding to a measurable 
section s : H\G — >■ G with s(if) = e. Let a G Ker(7r). Then, for every 
i G i 2 (ff\G,/C), we have 

o~(c(x,a))£,(xa) = £(x) for all x G H\G. 

Taking for £ mappings supported on a neighbourhood of if a, we see that 
a G if. Hence c(if, a) = a. Taking for £ continuous mappings with £(if) ^ 
and evaluating at if, we obtain that a G Ker(a). Since Ker(7i~) is normal in 
G, it follows that gag^ 1 G Ker(cr) for all g G G. 

Conversely, let a G G be such that gag^ 1 G Ker(cx) for all g G G. Since 

s(x)a = (s(x)as(x)~ 1 )s(x), 

we have c(x, a) = s(x)as(x)^ 1 for all x G H\G. Hence, for every £ G 
L 2 (H\G, K) and x G if \G, we have 

(ir(a)^)(x) = a(c(x,a))£(xa) = a(s(x)as(x)~ 1 )^(x) = £(x). 

This shows that a G Ker(7r) and the claim is proved. ■ 



7 Proof of Theorem [5 

Let T = V/A be a torus and if a countable subgroup of Aff (T) = Aut(T) xT. 
The implication (iii) (ii) is obvious and the implication (ii) (i) 
follows from |JuRo79] . The fact that (ii) implies (iii) has been proved in 
Theorem [HI Therefore, it remains to show that (i) implies (ii). Again by 
Theorem [HJ, it suffices to show that if the action of if on T has no spectral 
gap, then the same is true for the action of p&(H) on T, where p a is the 
projection from Aff(T) to Aut(T). This will be an immediate consequence of 
the next proposition. 

For a probability measure \i on Aff(T), we denote by p a (/i) the probability 
measure on Aut(T) which is the image of /i under p a . Let Uq be the Koopman 
representation of Aff(T) on Lq(T). 

Proposition 19 For every probability measure fi on Aff(T), we have 

\\U (m < \\U (pM)\\- 
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Proof Set T = Aut(T). Let T = Z d be the dual group of T. The Fourier 
transform sets up a unitary equivalence between Uq and the representation 

V of Aff (T) on f (f \ given by 

(*) V( 7 , a) X = xWx 7 for all X ef\ {1 T }, 7 G T, a G T, 

where x 7 £ ^ is defined by x 7 ^) — xlT" 1 ^))- 

Choose a set of representatives 5* for the T-orbits in T \ {It}- Then 
I 2 (t \ {It}) decomposes as the direct sum of Aff (T)-invariant subspaces 

t 2 (r\{i T })=@f(o x ), 

where O x is the orbit of x G S" under Y. 

It follows from Formula (*) above that the restriction V x of V to £ 2 (O x ) is 
equivalent to the induced representation Indr* J T x? where T x is the stabilizer 
of x m T and where x is the extension of x to T x ix T given by 

x(l, a ) = X(a) fo r all 7 e T x , a G T. 
The proposition will be proved if we can show that, for all x £ 5, we have 

(**) 11^)11 < ll^ x (Pa(/i))||. 

Now, the restriction of to T is equivalent to the natural representation 
of T in £ 2 (O x ), which is the induced representation Indp^J T lr- Observe that 

Indr^J T lr is equivalent to ^Indp^lr) °£>a- Hence, Inequality (**) follows from 

Herz's majoration principle (Proposition [T7I) and the proof of Theorem [5] is 
complete. ■ 

The following corollary gives a more precise information about the spec- 
tral structure of the Koopman representation associated to the action on T 
of a countable subgroup of Aff(T). 

Corollary 20 Let H be a a countable subgroup of Aff(T) and T = p a (H). 
There exists a Y -invariant torus factor T ofT such that the projection of H in 
Aff(T) is an amenable group and which is the largest one with this property: 
every other V -invariant torus factor SofT for which the projection of H in 
Aff (S) is amenable is a factor of T. Moreover, the torus factor T has the 
following properties: 
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(i) the projection ofY on Aut(T) is a virtually poly cyclic group; 

(ii) the restriction to L 2 (T)- L of the Koopman representation of H does not 
weakly contain the trivial representation 1#. 

Proof As for the proof of Theorem [5], we proceed by duality, using Fourier 
analysis and identifying V and A with their dual groups. 

Let Vrat(r) be the subspace generated by the union of T-invariant rational 
subspaces W of V for which IV is amenable. Then Kat(r) is a T- invariant 
rational subspace and, by Proposition EJ Y Vlz-t {v) is amenable. 

We claim that the natural unitary representation of Y on £ 2 (A\(V rat {Y) fl A)) 
does not weakly contain lp. Indeed, assume by contradiction that this is not 
the case. Then there exists a T-invariant mean m on A \ (V^ at (T) fl A)) . We 
consider the vector space V = V/V rait (Y) with the lattice A = p(A), where 
p : V — > V is the canonical projection. Then p*(m) is a T-invariant mean 
on A \ {0}. Hence, by Proposition [T3| there exists a non-trivial T-invariant 
rational W subspace of V such that the image of Y in GL{W) is amenable. 
Then W = p~ 1 (W) is a T-invariant rational subspace of V for which Yw is 
amenable. This is a contradiction since V ra ,t{Y) is a proper subspace of W. 

Let r° = rnZc(r) . By Proposition IT5l the eigenvalues of the restriction 
of every element in [r°, T ] to V Taut {Y) are all of modulus 1. Hence, by Corol- 
lary [161 the image of [r°,r°] in GL(V Tat (Y)) is virtually nilpotent. It follows 
that r^rr) is virtually polycyclic.B 

8 Some basic facts on Kirillov's theory and 
on decay of matrix coefficients of unitary 
representations 

We first recall some basic facts from Kirillov's theory of unitary representa- 
tions of nilpotent Lie groups. 

For a locally compact second countable group G, the unitary dual G 
of G is the set of classes (for unitary equivalence) of irreducible unitary 
representations of G. 

Let iV be a connected and simply connected nilpotent Lie group with Lie 
algebra n. Kirillov's theory provides a parametrization of TV in terms of the 
co-adjoint orbits in the dual space n* = HoniR(n, R) of n. We will review the 
basic features of this theory. 
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Fix I E n*. There exists a polarization m for I, that is, a Lie subalgebra m 
such that Z([m, m]) = and which is of maximal dimension; the codimension 
of m is \ dim(Ad* (iV)Z), where Ad*(iV)Z is the orbit of I under the co-adjoint 
representation Ad* of N. The induced representation Ind^xz is irreducible, 
where M = exp(m) and xi is the unitary character of M defined by 

Xi(exp X) = e 2nil{x \ XEm. 

The unitary equivalence class of Ind^x/ onr y depends on the co-adjoint orbit 
Ad*(iV)Z of I. We obtain in this way a mapping 

n*/Ad*(iV) N, 0^7i 

called the Kirillov mapping, from the orbit space n*/Ad*(iV) of the co-adjoint 
representation to the unitary dual N oi N The Kirillov mapping is in fact a 
bijection. For all of this, see |Kiri62] or |CoGr 89j. 

We have to recall a few general facts about decay of matrix coefficients 
of unitary group representations, following |HoMo79] and |Howe82j . 

Let (-7T, %) be a unitary representation of the locally compact group G. 
The projective kernel of ir is the normal subgroup P n of G defined by 

P n = {g EG : n(g) = K{g)I for some X w (g) E C}. 

Observe that the mapping g h-> A 7r ((?) defines a unitary character of P n . 
Observe also that, for £,1] EH, the absolute value of the matrix coefficient 

Cl v :g^(n(g)^ V ) 

is constant on cosets modulo P n . For a real number p with 1 < p < +oo, 
the representation 7r is said to be strongly LP modulo P n , if there is dense 
subspace D C %. such that, for every £, 77 E D, the function |C?J belongs 
to L P (G/P n ). Observe that then ir is strongly L q modulo P^ for any q > p, 
since CJ„ is bounded. 

Moreover, if tt is strongly L modulo P n , then ir is contained in an infinite 
multiple of Indp^A,,- (this can be shown by a straightforward adaptation of 
Proposition 1.2.3 in Chapter V of |HoTa92| ). 

We will also use the notion of a projective representation. Recall that a 
mapping n : G — > U (H) from G to the unitary group of the Hilbert space % 
is a projective representation of G if the following holds: 
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• ?r(e) = I, 



• for all gi,g2 G G, there exists c(gi,g 2 ) £ C such that 

^(9192) = c(g 1 ,g 2 )TT(g 1 )TT(g 2 ), 

• the function g i-> (n(g)^, 77) is measurable for all £,r) 

The mapping c : G x G -> S 1 is a 2-cocycle with values in the unit cercle 
S 1 . The projective kernel of 7r is defined in the same way as for an ordinary 
representation. Every projective unitary representation of G can be lifted to 
an ordinary unitary representation of a central extension of G (for all this, 
see |Mack76j or |Mack58j ). 

9 Decay of extensions of irreducible repre- 
sentations of nilpotent Lie groups 

Let N be a connected and simply connected nilpotent Lie group with Lie 
algebra n. 

The group Aut(A) of continuous automorphisms of N can be identified 
with the group Aut(n) of automorphisms of the Lie algebra n of N, by means 
of the mapping <p H- d e ip, where d e <p : n — > n is the differential of <p G Aut(iV) 
at the group unit. In this way, Aut(iV) becomes an algebraic subgroup of 
GL(n). Therefore, the group Aff (N) = Aut(A) K N of affine transformations 
of N is also an algebraic group over R. 

Set G := Aff (A). In the following, we view A as a normal subgroup of G. 
The group G acts by inner automorphisms on iV and hence by automorphisms 
on n, n*, and iV; observe that, for g 6 G and / 6 n*, we have 

(Ad* (n)l) 9 = Ad* {gng- 1 ) (I 9 ) for all n G N. 

This shows that g permutes the orbits of the co-adjoint representation, map- 
ping the orbit of / onto the orbit of I 9 . Let it G iV with corresponding co- 
adjoint orbit O. The representation n 9 G N, defined by 7T 9 (n) = nlgng^ 1 ), 
corresponds to the orbit O 9 . 

For a co-adjoint orbit O in n*, we denote by Go the stabilizer of O in G. 
Similarly, 

G w = {g G G : n 9 is equivalent to n} 
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is the stabilizer in G of tt G N. Observe that, if tt is the representation 
corresponding to the co-adjoint orbit O in Kirillov's picture, then = Go- 
Observe also that iV is contained in G n . 

The following elementary fact will be crucial for the sequel. 

Proposition 21 Let tt be an irreducible unitary representation of N. The 
stabilizer G n of tt is an algebraic subgroup of G. Moreover, for every I in 
the co-adjoint orbit corresponding to tt, we have G^ = GiN where Gi is the 
stabilizer of I in G 

Proof The co-adjoint orbit O associated to tt is an algebraic subvariety of 
n* (see Theorem 3.1.4 in |CoGr 89j). It follows that G n = Go is an algebraic 
subgroup of G. Moreover, since N acts transitively on O, it is clear that 
G = G t N for every leO.M 

Let tt be an irreducible unitary representation of N, with Hilbert space 
%. It is a well-known part of Mackey's theory of unitary representations of 
group extensions that there exists a projective unitary representation tt of 
G w on % which extends tt. Indeed, for every g G G n , there exists a unitary 
operator Tr(g) on 7-L such that 

Tr{g{n)) = TT(g)iT(n)TT(g)^ 1 for all n E N. 

One can choose Tr{g) such that g i— > 7r(g) is a projective representation unitary 
representation of G^ which extends tt (see Theorem 8.2 in |Mack58] ) . 

The following proposition, which will play a central role in our proofs, is 
a consequence of arguments from |HoMo79] concerning decay properties of 
unitary representations of algebraic groups. 

Proposition 22 Let tt be an irreducible unitary representation of N on Tt 
and let tt be a projective unitary representation of G n which extends tt. There 
exists a real number p > 1, only depending on the dimension ofG, such that 
tt is strongly L p modulo its projective kernel. 

Proof Since tt is irreducible, Tr{g) is uniquely determined up to a scalar 
multiple of the identity operator / for every g G G n . In particular, all projec- 
tive unitary representations of G^ which extend tt have the same projective 
kernel. 

We will need to give an explicit construction of a projective representation 
of G n extending tt. This representation will lift to an ordinary representation 
of a two-fold cover of CL. 
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We denote by O the co-adjoint orbit associated to it and we fix throughout 
the proof a linear functional I in O. 

Set H = Aut(iV) so that G = H x N. Let H be the stabilizer of I in H. As 
shown in Proposition |2H GV is an algebraic subgroup of G and = HiN. It 
is clear that Hi is also an algebraic subgroup of G. Let Ui be the unipotent 
radical of Hi. Then [/ = UiN is the unipotent radical of G n . 

•First step: We claim that tt can be extended to an ordinary unitary repre- 
sentation a of U. 

Indeed, let U/ be the Lie algebra of U\. We extend I to a linear functional 
I on the Lie algebra u = Ui © n of £/ by defining = for all X G Uj. 

Let m C n be a polarization for Z. We claim that m := © m is a polar- 
ization for I. Indeed, we have Z([rn, m]) = since [X, Y] G n and (expX)Z = I 
for all X G U/ and Y G m. Moreover, the codimension of fii in u coincides 
with the codimension of m in n and the dimension of the co-adjoint orbit of I 
under Ad*(U) coincides with the dimension of Ad* (N)l. Since the codimen- 
sion of m in n* is ~ dim(Ad*(A^)/), it follows that the codimension of m in u* 

is \ dim( Ad* (U)l). Hence, m is a polarization for I. 

Recall that ir is unitarily equivalent to the induced representation Ind^Xh 
where M = exp(m) and \i is the unitary character of M defined by 

Xi(expX) = e 2wil(x) for all X G m. 

Let M be_the closed subgroup of U corresponding to m. The unitary charac- 
ter xj °f M given by I coincides with \i on M. Since a fundamental domain 
for M\N is also a fundamental domain for M\U, we see that Ind~x^ can 
be realized on the Hilbert space of Ind^x; an d that a := Ind~xj- extends 
tt = Ind^x«- 

• Second step: We claim that G a = G n . 

It is obvious that G a C G n . Let Hi = RUi be a Levi decomposition of 
Hi, where R is a reductive subgroup of Gi. In order to show that G n C G a , 
it suffices to prove that R C G a , since = RU. Now, R leaves U/ and n 
invariant and fixes I. Hence, R fixes the extension I of I defined above and 
the claim follows. 

• Coda: As a result, upon replacing N by U, we can assume that N is the 
unipotent radical of G n . Since the connected component of G n has finite 
index, we can also assume that G n is connected. 
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As shown above, we have a Levi decomposition G n = RN with R a 
reductive subgroup contained in G\. According to |Howe73j . we can find in 
iV algebraic subgroups K\ C P\ C N\ with the following properties: 

• Ki, Pi, and N% are normalized by R; 

• K\ and P\ are normal in N% and Nx/K\ is a Heisenberg group with 
centre P\jK\\ 

• there exists a unitary character A of P\jK\ such that 7r is equivalent 
to the induced representation Ind^vrx, where tti is the lift to N% of the 
unique irreducible representation of the Heisenberg group N\/K\ with 
central character A. 

The action of R on N\ / K\ defines a homomorphism from R to the symplectic 
group Sp(N\/P\) of the vector space N\/P\] as a result, we have a homomor- 
phism yj : RNi -» Sp{Nx/P x ) x {Nx/Kx). The representation tti of iVi/^i 
extends to a projective representation u; of Sp{N\/P\) k (Ni/Ki), called the 
metaplectic (or oscillator, or Shale-Weil) representation; more precisely, there 
exists a two-fold cover Sp of Sp(Ni/P\) and a unitary representation u; of 
Sp x (Ni/Ki) on the Hilbert space of 7Ti which extends 7Ti. 

We can lift <p to a homomorphism <£> : iL/Vi — >■ Sp x (Ni/Ki) for a two- 
fold cover of R. Then p := w o if is a unitary representation of RN% on the 
Hilbert space of tt\ which extends tti. 

Set 7? := IndfjJ p. Then 7r is a unitary representation of the two-fold 

cover G n := RN of G n = RN; moreover, 7? extends n, since 7r is equivalent 
to Ind^ 7Ti, and p extends n\. 

Observe that G n is in general not an algebraic group. Let p : G w — > G w be 
the covering map. Let us say that a connected subgroup H of G w is reductive 
if is a reductive subgroup of G n . We claim that has no non-trivial 
reductive normal subgroup. Indeed, let H be a reductive normal subgroup 
of G n . Since = i?iV is a Levi decomposition of G n , the normal subgroup 
p(-ff) of G n is conjugate to a subgroup of i? and therefore p(H) C -R. Hence, 
p(H) centralizes N. It follows that p(H) is trivial since p(H) C Aut(iV). 

Now, the same arguments as those on pages 87-93 in [HoMo79] show 
that there exists an integer k such that the fc-fold tensor power 7r® fc of 7r is 
square integrable modulo the projective kernel P% of tt. For instance, let us 
check how the first step in [HoMo79"] towards this claim carries over to our 
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situation. For an integer k, we are interested in the tensor power 7r . In order 
to apply Mackey's tensor product theorem (see [Mack 76 [ Theorem 3.6]), we 
have to show that (RNi) k and the diagonal subgroup AG n of G k are regularly 
related. Now, the quotient space G k / \RN\) k is can be canonically identified 
with G k r /(RNi) k 1 and the action of AG V on G k /(RNi) k corresponds, via the 
covering mapping p : G n — > G n , to the action of AG n on G k /(RN 1 ) k . Since 
AG n of G k are algebraic subgroups of G k , the claim follows. ■ 

Remark 23 According to [HoMo79"l p. 93], a crude bound for the number p 
in Proposition [221 is 

p < (dim(Gy + l) 2 . 

The generalized metaplectic representation tt which appears in the proof 
above has been studied by several authors (see |Dufl72] . |Howe73j . [Lion79j ). 

10 Rational unitary representations of a nilpo- 
tent Lie group 

As in the previous section, let N be a connected and simply connected nilpo- 
tent Lie group and 

G := Aff(iV) = Aut(iV) x N. 

Let tt be an irreducible unitary representation of N and G n the stabilizer 
of 7r in G. Let it be a projective unitary representation of G n extending tt. In 
the following proposition, we describe the projective kernel of tt. 

Proposition 24 Let L w be the connected component of Ker(7r). Set N = 
N/Lk and let p : N —¥ N be the canonical projection. For g = (h,n) E G^ 
with h G Aut(iV) and n G N, the following are conditions are equivalent: 

(i) 9 e ft; 

(ii) h leaves L v invariant and the automorphism of N induced by h coin- 
cides with the inner automorphism Ad(p(n) _1 ). 

Proof Assume that g = (h, n) G P%- By definition of P^, we have tt (h) = 
A 7 r(^)7r(n _1 ). It follows that, for every x £ N 

ir(h(x)) = tt \h)TT \x)tt (h) = 7r(n _1 )7r(a;)7r(n) = Tr(n~ 1 xn), 



27 



that is, 

h(x)n~ 1 x~ 1 n G Ker(7r) 
Since iV is connected, this is equivalent to 

h(x)n~ 1 x~ 1 n G L n i 



for all x G N. 



for all x G N. 



As L n is normal in N, this shows that L v is invariant under h and that the 
automorphism induced by h on A" is Ad(p(n) _1 ). 

Conversely, suppose that is invariant under /i and that the automo- 
morphism h induced by h on iV coincides with Ad(p(n) _1 ). Observe that 
ii factorizes to a representation a of N. Let a be an extension of a to the 
stabilizer of a in Aut(AT) x iV. Then 

a{h)a{p{x))a{hY 1 = a{p{n))^ 1 a{p{x))a{p{n)) for all x <E N, 

that is, a(p(n))a(h) commutes with a(p(x)) for all x G iV. Since 7r is irre- 
ducible, it follows that a(p(n))a(h) and hence Ti{n)Tr{h) is a scalar operator. 
This means that g = (h, n) G -P^.B 

Next, we review some well-known facts about rational structures on n 



Recall first that a lattice T in a locally compact group G is a discrete 
subgroup such that the translation invariant measure induced by a Haar 
measure on G on the homogeneous space T\G is finite. 

The Lie algebra n (or the corresponding nilpotent Lie group N = exp(n)) 
has a rational structure if there is a Lie algebra Uq over Q such that n = 
n.Q ®q R. If n has a rational structure given by uq, then iV contains a 
cocompact lattice A such that log A C Uq. Conversely, if A^ contains a 
lattice A, then A is cocompact and n has a rational structure given by riQ = 
Q — span(log A). 

Assume from now on that A^ has a rational structure riQ and let A be 
a lattice inducing this rational structure. We say that a R-subspace t) of 
n is rational if f) = R — span(f) D nq). All subalgebras in the ascending or 
ascending series as well as the centre of n are rational. A connected closed 
subgroup H of A^ is said to be rational if the corresponding subalgebra Lie 
algebra f) is rational. This is equivalent to the fact that H fl A is a lattice in 



Let if be a rational connected normal closed subgroup of A^ with Lie 
algebra fj Then N/H has a canonical rational structure (tl/fy)q induced by 
the lattice AH/H of N/H. 



(see |CoCr89j . |Ragh72| ). 



H. 



28 



There is a unique rational structure Uq on the dual space n* defined as 
follows: a functional I G n* belongs to Uq if and only if l(X) G Q for all 
X G n Q . 

An important role will be played later (in Section fT2l) by irreducible uni- 
tary representations of iV which are rational in the sense of the following 
definition. 

Definition 25 An irreducible unitary representation tt of iV is rational if its 
co-adjoint orbit O n is rational, that is, if O n fl riq 7^ 0. 

We fix for the rest of this section a rational irreducible unitary represen- 
tation 7T of N. 

We first establish the rationality of the kernel of 71". 

Proposition 26 The connected component L n of Ker(7r) is a rational nor- 
mal subgroup of N. As a consequence, A = AL n /L n is a lattice in N/L n . 

Proof Since ir is rational, the corresponding co-adjoint orbit in n* contains 
a functional I G Uq. The representation it is unitarily equivalent to Ind^Xh 
where m is a polarization for I, M = exp(m), and xi is the unitary character 
of M corresponding to I. 

Recall from Lemma HE] that Ker(7r) coincides with the largest normal 
subgroup of iV contained in Ker(x/). For the ideal [ corresponding to Ker(7r), 
we have therefore 

1= p| Ker(Ad*(n)/) = f] Ker(Ad*(exp X)l). 

Since Ker(Ad*(exp X)l) is rational for all X G Uq, it follows that I is rational. 
Thus, the connected component L n of Ker(7r) is rational, by definition. ■ 

The set Aut(A\iV) consisting of the automorphisms 7 G Aut(A) with 
7(A) = A is a discrete subgroup of the algebraic group Aut(AT). 

Let G v be the stabilizer of n in G and tt a projective unitary representation 
of G n extending n. Set 

T n = G^ n Aut(A\A). 

The projective kernel P% of tc was determined in Proposition [2H . We will 
need to have a precise description of fl (T^ x N). 
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As before, let L n be the connected component of Ker(7r), iV = N/L w , 
p : A — )• A the canonical projection, and A = p(A). Observe that g(L n ) = L n 
for all g G fl Aut(A). Consider the induced continuous homomorphism 

<p : G n -> Aff(A) = Aut(A) x A. 

Proposition 27 Let Norm(A) 6e t/te normalizer of A in A. 

(i) We have 

P„ n (IV K iV) = y?- 1 ({(Ad^ar 1 ) : x G Norm(A)}) . 

(ii) Let A := {(Ad(x), x~ l z) : x G A, z G Z(A)} ; w/iere Z(JV) £/ie 
centre of N. Then (f~ 1 (A) fl (r„. x A) is a subgroup of finite index in 

p„ n (r, x a). 

Proof (i) By Proposition 1241 we have 

P~ = tp-l ({(Ad(x),x^) :xeN}). 

Let g = (7, n) G fl (IV x A). Then yj(g) = (Ad(x), a; -1 ) for some x G A. 
Since 7(A) = A, we have Ad(x)(A) = A, that is, x G Norm(A). Conversely, 
it is obvious that, if g = (Ad(x),x _1 ) for some x G Norm(A), then g G 

p^n^xA). _ _ 

(ii) In view of (i), it suffices to prove that the subgroup AZ(N) has finite 
index in Norm (A). 

To show this, recall that A is a cocompact lattice in A (Proposition 126]) . 
Let Norm(A)o be the connected component of Norm(A). Since Norm(A)o 
normalizes A and since A is discrete, Norm(A)o lies in the centralizer of 
every element of A. As A is Zariski dense in A (see e.g. Theorem 2.1 in 
Ragh72| ), it follows that Norm(A) = Z(N). Since the projection of A has 
finite covolume in the discrete group Norm(A) /Norm(A) , the claim follows. ■ 

The next proposition will allow us to deduce decay properties of repre- 
sentations of G n restricted to x A. 

Proposition 28 The subgroup (T n x N)P^ is closed in GV. 
Proof Using Proposition [231 we see that 

p~N = cp' 1 (Ad (A) x A) 
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and hence 

(r w x N)P^ = ip- 1 ((<p(T w )Ad(N)) k N) . 

It therefore suffices to show that ip(T n )Ad(N) is closed in Aut(iV). 

Observe that, for every 7 G T n , we have 7(A) = A (since C Aut(A\iV)) 
and hence (f{T^) C Aut(A\iV). 

Let (7i)j and be sequences in 1^ and in Ad(AQ such that 

\xm(p("/i)xi = g e Aut(AT). 

i 

Since Ad(A) is a cocompact lattice in Ad(N), there exists a compact subset 
D of Ad(AT) such that X{ = 5idi for some Si G Ad(A) and di E D. As D is 
compact, we can assume that lim<ij = d G Ad(AQ exists. Then linij (pi^jijSi = 
gd~ l . Now, 

Ad (A) = <p(Ad(A)) c ^(r.) 

and ^(Ttt) is a subgroup of the discrete group Aut(A\AQ. It follows that 
gd -1 G (^(r^), that is, g G (p(r, r )Ad(A^). Hence, v?(r 7r )Ad(A^) is closed in 

n. m 

Corollary 29 Lei A = {(Ad(x), x~ x z) :iGA,z6 Z(iV)} and <^ : -> 
Aff(A^) i/ie canonical projection, where N = N/L n . The restriction ofn to 
k iV is strongly LP modulo y> _1 (A) fl (IV x AT) /or £/ie rea/ number p 
appearing in Proposition BH 

Proof We know from Proposition 1271 that <£> _1 (A) fl (r w x N) has finite 
index in P% fl (r„- ix N). Hence, it suffices to prove that the restriction of tt 
to k iV is strongly LP modulo P% D (1^ x N). 

By Proposition Eg (rv x N)P^ is closed in G? w . Therefore, {T n x N)P^/P^ 
is homeomorphic as a (r„- x A^)-space to (r^ x N)/ (P% fl (r^ X AT)). It follows 
from Proposition [22] (see the proof of Proposition 6.2 in [HoMo79] ) that the 
restriction of n to \x N is strongly LP modulo P^ fl (r^ x N). ■ 

11 A general estimate for norms of convolu- 
tion operators 

Let G be a locally compact group. For a unitary representation (71", %) of 
G, the contragredient (or conjugate) representation W acts on the conjugate 
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Hilbert space %. Recall that, for an integer k > 1, the /c-fold tensor product 
ir® k of 7r is a unitary representation of G acting on the tensor product Hilbert 
space 'H m . 

We will need in a crucial way the following estimate which appears in the 
proof of Theorem 1 in |Nevo98] . 

Proposition 30 Let fi be a probability measure on the Borel subsets of G. 
Let (tt, %) be a unitary representation of G . For every integer k > 1, we have 

Ik^U < || (n®Jff k Ml 1 /* 

Proof Denote by ft the probability measure on G defined by £i(A) = 
n(A~ l ) for every Borel subset A of G. 

Using Jensen's inequality, we have for every vector £ 6 H, 



G 



21,- 



M#)6,6)d(/2*/i)(#) 



< 



G 



G 



G 



\{(n®wr k (ii*m®t;r\(z®t;r k )\ 

||(7r®7fO«*(A*)(e®0®*ir- 



and the claim follows. 



12 Analysis of the Koopman representation 
of the affine group of a nilmanifold 

Let N be a connected and simply connected nilpotent Lie group, A a lattice 
in N. There is a unique translation invariant probability measure z^a\at on 
A\N and it is induced by a Haar measure on N. This measure is also invariant 
under Aut(A\AT). 
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We fix throughout this section a subgroup T of Aut(A\iV). The Koopman 
representation U of V x A associated to the action of V x A on A\N is given 
by 

U(7,n)£(x) = £(7 -1 (aM 7 6 T, n G A, £ G L 2 (A\A), z G A\A. 
In particular, we have 

(1) Ui^-^Uir^Uin) = C/"( 7 - 1 (7i)) for all 7 G T, n G A. 

Recall that T = A[iV, A]\A is the maximal factor torus associated to 
A\A. The action of Aff(A\A) on A\N induces an action of Aff (A\A) on T. 
We identify L 2 (T) with a closed subspace of L 2 (A\A). 

More generally, let L be a connected closed subgroup of A which is both 
rational and invariant under T. Then AflL is a lattice in L and A = AL/L is a 
lattice in N = N/ L. There is an induced action of T k N on the subnilmanifold 
L/(A n L) and on the factor nilmanifold A\N. The canonical mapping p : 
A\N 1 — y A\N is T x A-equivariant and presents A\A as a fibre bundle over 
A\N with fibres diffeomorphic to L/(A n L). The Hilbert space L 2 (A\A) 
can be identified, as V x A-representation, with the V x A-invariant closed 
subspace of L 2 (A\A) consisting of the square- integrable functions on A\A 
which are constant on the fibres of p. 

We write 

L 2 (A\A) = L 2 {T)@H, 

where H, is the orthogonal complement of L 2 (T) on L 2 (A\A), and observe 
that "H is invariant under Aff(A\A). 

We are going to show that the restriction of U to % has a canonical de- 
composition into a direct sum of induced representations from the stabilizers 
in r ix A of certain representations 7r G A; this decomposition can be viewed 
as generalization of the decomposition of L 2 (T) which appears in the proof 
of Proposition [THJ 

Since A is cocompact in A, we can consider the decomposition of % into 
its A- isotypical components: we have 

ttSS 

where S is a certain set of infinite-dimensional pairwise non-equivalent irre- 
ducible unitary representations of A; for every n G S, the space Ti^ is the 
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union of the closed [/(iV)-invariant subspaces K of H for which the corre- 
sponding representation of N in K. is equivalent to tt. According to |Moor65[ 
Corollary2], every n G £ is rational in the sense of Section [TOJ Every H„ is a 
direct sum of finitely many irreducible unitary representations; therefore, the 
restriction of U (N) to % n is unitarily equivalent to a tensor product n £g> / 
acting on K, n ®£ n , where K, n is the Hilbert space of tt and where C w is a finite 
dimensional Hilbert space. (For a precise computation of the dimension of 
jC n , see |Howe71] and |Rich71] ; the fact that C n is finite-dimensional will not 
be relevant for our arguments.) 

Let 7 be a fixed automorphism in T. Let U 1 be the conjugate represen- 
tation of U by 7, that is, U"'(g) = U(j~ 1 (g)) for all g G G. On the one 
hand, for every tt G S, the subspace H ^-i is the iso typical component of 
[/ 7 |at corresponding to 7T. On the other hand, relation (1) shows that U{^ 1 ) 
provides a unitary equivalence between U\n and £/ 7 |at. It follows that 

U{ 1 - 1 ){U 1 <)='H^-x for all 7 GT 

In summary, we see that V permutes the "H^'s among themselves according 
to its action on N . 

Write £ = \J ieJ where the Ej's are the T-orbits in S, and set 

Every is invariant under 1^ x N and we have an orthogonal decomposition 

i 

Fix i G /. Choose a representation 7Tj in Sj and set = 7^^. Let Tj denote 
the stabilizer of 7Tj in T. The space Hi is invariant under Tj x N. Let be 
the corresponding representation of Tj x N on 

Choose a set of representatives for the cosets in 

r/r, = (r k jv)/(r, x iV) 

with e G 5*i. Then Sj = {ixf : s G Si} and the Hilbert space is the sum 
of mutually orthogonal spaces: 

sG5i 
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Moreover, Hf is the image under U(s) of for every s G Si. This exactly 
means that the restriction Ui of U to of the Koopman representation U 
of T x is equivalent to the induced representation Ind£*^Vi. 

As we have seen above, we can assume that "Hi is the tensor product 

Hi = lC t ® d 

of the Hilbert space /Q of 7Tj with a finite dimensional Hilbert space £j, in 
such a way that 

(2) Vi(n) = 7Tj(n) <g> 7/v for all n e N. 
Let g G Tj x N. By (1) and (2) above, we have 

(3) V i (g)(ir i (n)®I Ci )V i (g)- 1 = 'iri(gng- 1 )®Ic i for all n e N. 

On the other hand, let Cr, be the stabilizer of 7Tj in Aff (Af); then 7Tj extends to 
an irreducible projective representation 7Tj of Gi (see the remark just before 
Proposition |2"2"|) . Since 

^i(g)M n ) ^i(9~ X ) = iXi(gng- 1 ) for all n e N, 

it follows from (3) that the operator (^(p -1 ) <g) IcJ Vi(<7) commutes with 
7Tj(n) ® for all n <E N. Since 7Tj is irreducible, there exists a unitary 
operator Wi(g) on £j such that 

V i (g)=7F i (g)®W i (g). 

It is clear that Wi is a projective unitary representation of Tj x AT, since V{ 
is a unitary representation of Tj k AT. 

13 Proof of Theorem [D first step 

We summarize the discussion from the previous section. We have a first 
orthogonal decomposition into Aff (A\AT)-invariant subspaces 

L 2 (A\A0 = L 2 (T)®U, 
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where T is the maximal torus factor of A\N. Let T be a subgroup of Aut(A\iV). 
There exists a sequence of T-invariant sets (Ej)j g j of rational infinite dimen- 
sional unitary irreducible representations of iV such that we have a decom- 
position into mutually orthogonal F x AMnvariant subspaces 

iei 

with the following property: for every i, the representation U{ of T x N 
defined on is equivalent to 

where 7Tj is a representation from where 7Tj is the restriction to I\ ix A" of 
an extension of 7Tj to the stabilizer Cr, of 7Tj in G = Aff(AT), and where Wj is 
some finite dimensional projective unitary representation of Tj k AT. 

We need to recall the decomposition of the representation U tOT of Y on 
Lq(T) from Section [3 Let T = Z d be the dual group of T and let S be a set 
of representatives for the T-orbits in T \ {It}- Then 

(4) £/tor = 0Ar/r x , 

where T x is the stabilizer of x i n T and Ar/r is the natural representation 

of r on^ 2 (r/r x ). 

In the following result, we establish a link between the restrictions to % 
and to Lq(T) of the Koopman representation of T. This result, which is a 
consequence of the discussion above and of results from Section[TDl is a major 
step in our proof of Theorem [TJ 

Recall that p a denotes the canonical projection A&(A\N) —> Aut(A\A"). 
For a probability measure [i on Aff(A\AQ, let p a (A*) be the probability mea- 
sure on Aut(A\AQ which is the image of /i under p a . 

Proposition 31 There exists an integer k > 1 only depending on dim A" 
with the following property. Let T be a subgroup of Aut(A\AT) which stabi- 
lizes some 7i E N appearing in the decomposition H = ©„. e2 of H into 
isotypical components under N. For every probability measure p, onT \x N, 
we have 

\\UM)\\ < \\Utor( P M)\\ l/2 \ 

where U n and U tm are the restrictions of the Koopman representation ofTtxN 
to %-k and Lq(T) respectively. 
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Proof Let G n be the stabilizer of 7r in G = Aff(iV). Let tt a projective 
representation of G n extending 7r. 

As we have seen above, U n is equivalent to (ttItkn) <S> W for some finite 
dimensional projective unitary representation W of T ix N. Let P denote the 
projective kernel of U n . Observe that P — Pi fl P2, where Pi and P2 are the 
projective kernels of 7r|r><7v and W . 

Denote by L v the connected component of Ker(7r) and iV = N/L n . As in 
Section [TUJ let ip : G^ — > Aff(A) be the corresponding homomorphism and 

A = {(Ad(a;), x^z) : x G A, z G Z(N)}, 

where A is the lattice AL n /L n in N and Z(N) the centre of N. Then 

Q:=<p-\A)n(T v x N) 

is a subgroup of finite index of Pi (Proposition [27]). By Corollary [29j there 
exists a real number p > 1 only depending on the dimension of Aut(A) tx A 
such that 7f|r^t<Af is strongly L p modulo Q. 

We claim that Q is contained in P. Indeed, for g e Q,we have 

<p(g) = (Adix)^-^) 

for some x E A and z G Z(N). Hence ac ^s as the right translation by 
z on L 2 (A\N). Observe that is contained in L 2 (A\N) and that g acts as 
'p(g) on Htt. Since N acts as a multiple of the irreducible representation ix 
on H^, it follows that g G P and the claim is proved 

As a consequence, we see that Q is a subgroup of finite index in P. 
Observe that Q is also contained in P 2 . It follows that £7^ = (7r|r><7v) <8> is 
strongly L p modulo Q and hence U n is strongly L p modulo P. 

Let A; be an integer with k > p/A. Then the tensor power (U w ® U n ^ k 

is strongly L 2 modulo P. Hence, as discussed in Section [8j (P^ eg) U^)^ is 
contained in an infinite multiple of the induced representation Indp^A,,-, for 
the associated unitary character A„- of P. It follows that, for every probability 
measure fi on T ix N, we have 

|| (U v ®Uj* k (ji)\\<\\ (ind^A*) (/i)|| 
and hence, using Proposition [301 

\\UM\\ < || (ind^Aj (a,)!! 1 /* 
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On the other hand, observe that PN = p. A l {p&{P)) is closed in Aff (A\N), 
as Aut(A\AQ is discrete. Since, by induction by stages, 

Ind^A^ = lnd% N (ind^A,) , 

we have, using by Herz's majoration principle (Proposition [T7|) . 

|| (lndp KA %) (fj)\\ < ||A ( r K iV)/Piv(^)||- 

Now, \(rxN)/PN = (Ar/ Pa (p)) ° p a and hence 

H-Vkao/paKaOH = Pr/pa(P)(Pa(^))||- 

As a consequence, the proposition will be proved if we establish the following 
inequality 

(5) l|Ar/ Pa (P)(Pa(X))|| < ||^tor(Pa(^))||- 

To show this, recall (see (4) above) that Utor is equivalent to the direct sum 
© xe 5 ^r/r x , where S is set of representatives for the T-orbits in T \ {It}- As 
a consequence, Inequality (5) will be proved if we can show that there exists 
X € T \ {It} such that 

HAr/p a (P)(PaO"))H < HAr/r x (/i)||. 

By Herz's majoration principle again, it suffices to show that exists x £ T 
with x 7^ It such that p a (P) C T x . For this, recall that, for every g E P <Z P\, 
there exists x E N such that 7 = p a (flO acts as Ad(x) on A" (Proposition |27|) . 
For every unitary character x of N, we have 

X(^(7)(y)) = Xixyx' 1 ) = x(y) for all y 6 JV. 

Thus, Pa,(P) fixes every unitary character of A". 

Observe that A" is non-trivial, since ir 7^ ljy. Choose a non-trivial unitary 
character of A" which is constant on the cosets of A and denote again by x 
its lift to N. Then ^ eT \ {lp} and x is fixed by Pa,(P). ■ 

Remark 32 With Remark [23], we see that a rough estimate for the integer 
k appearing in the statement of Proposition EH is 

k < ^ (dim (Aut(iV) x iV) + l) 2 + 1 < ^((dim(A^)) 3 + l) 2 + 1. 



38 



Example 33 Let N = H 2n +i(R) be the (2n + l)-dimensional Heisenberg 
group (over R) and let A be a lattice in N. Then Aut(A\iV) contains a sub- 
group of finite index T consisting of automorphisms which fix every infinite 
dimensional representation n & N (see |Foll89j ). Let if be a countable sub- 
group of Aff(A\A r ). Assume that the action of H on A\N does not have a 
spectral gap. It follows from Proposition ED that there is a subgroup Hi of 
finite index in H, such that the action of p a (Hi) on T does not have a spectral 
gap. Therefore, using Theorem [5] the action of Hi and hence of the action 
of H on T does not have a spectral gap. This result generalizes Theorem 3 
in [BeHelOj to groups of affine transformations of Heisenberg nilmanifolds. 

14 Proof of Theorem [D completion of the 
proof 

We are now in position to give the proof of Theorem [TJ In view of Theorem [5] 
we only need show that (ii) implies (i). 

Let if be a countable subgroup of Aff(A\iV). Assume, by contraposition, 
that the action of H on A\N does not have a spectral gap. We have to prove 
that the action of H on T does not have a spectral gap. 

Set T = p a (H). By Theorem EJ it suffices to prove that the action on T of 
some subgroup of finite index in T does not have a spectral gap. Let U H be 
the representation of Aff(A\iV) on the orthogonal complement H of L 2 (T) in 
L 2 (A\N) and Ut ov the representation on Lq(T). Our theorem will be proved 
if we can show the following 

Claim: Let \x be an aperiodic measure on H. Assume that ||^(/i)|| = 
1. Then there exists a subgroup A of finite index in T and an aperiodic 
probability measure v on A such that ||f/tor(Pa( l/ )) || = 1- 

To prove this claim, we proceed by induction on the dimension of the 
Zariski closure Zc(T) of T in Aut(iV). 

If dimZc(r) = 0, then T is finite and there is nothing to prove. 

Assume that dimZc(r) > 1 and that the claim above is proved for 
every countable subgroup of Hi of AS(A\N) for which dimZc(p a (i?i)) < 
dim Zc(r). 

Recall from Sections [12] and [13] that, as T k A^-representation, U H is 
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equivalent to a direct sum 



where Tj is the stabilizer in T of a rational representation 7Tj G iV and Vi is a 
unitary representation of Tj x N. 

Let Jfi n C / be the set of all i G / such that Tj has finite index in T and 
set loo = / \ J fin . Let 

C/ fln = Ind^y, and = Ind^y, 

and denote by Hfi n and Hoo the corresponding subspaces of defined re- 
spectively by £/fin and 6^0 . Since ||{7 w (/x)|| = 1, two cases can occur. 
• First case: we have ||£/oo(aOH — 1- By Herz's majoration principle, we have 

||( Ind rS^) 0)|| < ||VkJV)/(T(kjv)(^)|| 
for every i G J fin . Since A^ao/o^ao = A r / r . op a , it follows that 



Xr/ViipM) 



1. 



Let £ > 0. We can choose i 6 such that 



(6). 



We claim that dimZcfTj) < dimZc(r). Indeed, otherwise Zc(r^) and 
Zc(r) would have the same connected component C°, since Zc(I\) C Zc(T). 
As the stabilizer of 7Tj in Aut(iV) is Zariski closed (Proposition I2TT) . C° would 
stabilize tt%. Therefore, TdC would be contained in Tj. But TnC has finite 
index in T. Hence, Tj would have a finite index in T and this would be a 
contradiction, since i G 1^. 

Let /ij be a probability measure with support equal to (Tj xN)nH. Then 
(/ij + fj)/2 is an aperiodic probability measure on H. Since ||f/ w (/i)|| = 1, 
we also have \\U H ({ni + /i)/2)|| = 1. Therefore, ||f/ w (/ii)|| = 1. Since 
dim Zc(Tj) < dim Zc(T), it follows from the induction hypothesis that \\U tor (ni] 
1. Then, by Theorem El we also have ||{/tor(Pa(/-0)ll = 1- 
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On the other hand, recall from (4) that, replacing T by Tj, the IV 
representation U tOT decomposes into a direct sum 



U tox = A ri /(r x nr 8 )- 
xes* 



As a consequence, we have 



0(Ar l /(r x nr l ))(Pa(/ii)) 



1. 



Observe that p a (A*i) is an aperiodic probability measure on I\ (in fact, the 
support of p a ([ii) is It follows that the Tj-representation © xgS Ary(r x nr;) 
weakly contains the trivial representation lr r Since 

^drvir, = A r /r, and Ind r .A r< /(r x nri) = A r /(r x nr\) 



it follows, by continuity of induction (see Proposition F.3.5 in [BeH V08] ) . 
that the T-representation © xeS Ar/(r x nr\) weakly contains Ar/rv As a con- 
sequence, we have 



||A r/r ,(p a (//))|| < 



0(Ar/(r x nr l ))(Pa(^)) 

xe5 



Observe that, by Herz's majoration principle again, we have 



l*r/(r x nr,(Pa(/i))|| < ||A r /r x (Pa(/x))||. 



Hence 



0A r/ r x (Pa(/i)) 

x&s 



Using Inequality (6), it follows that 

ll^tor(Pa(/i))|| > 1-6. 

Since this is true for every e > 0, we obtain that ||f/tor(Pa(/-0)|| 
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• Second case: we have ||L/fi n (/i)|| = 1. By the Noetherian property of the 
Zariski topology on Aut(iV), we can find finitely many indices i\, . . . ,i r in 
J nn such that 

Zc(r il )n---nzc(rv) = f| Zc(r,). 

Since stabilizers of irreducible representations of N are algebraic (Proposi- 
tion [21]), the subgroup A := fl • ■ ■ fl T ir stabilizes 7Tj for every i <E Ia n . 
Moreover, A has finite index in T, since every Tj has finite index in T. 

From Sections [12] and [131 we have a decomposition of into A x N- 
invariant subspaces 

where Hi is the isotypical component corresponding to 7Tj under the action of 
N. Let v be a probability measure with support equal to (A x N) fl H. Con- 
sidering as above the aperiodic measure (/z + i/)/2 on H, we have ||£7 n n(^))|| = 
1, since ||[/ fln (//)|| = 1. 

On the other hand, by Proposition EH there exists an integer k > 1, 
which is independent of i, such that 

\\Ui(v))\\ < \\U tor (P^))\\ 1/2k for all i E J fin 
where Ui is the representation of A ix iV on Hi. As a consequence, we have 

||£/fi»)|| < ||C/to r (Pa(^))H 1/2fc 

and it follows that ||^tor(p a (^)) || = 1- Since the support of p a (v) is the sub- 
group A of finite index in T, this completes the proof of Theorem [TJ ■ 

Remark 34 The proof of Theorem [1] we gave above is not effective: it does 
not give, for a probability measure fi on Aut(A\iV), a bound for the norm 
of n under U H in terms of the norm of \x under [/tor and/or other "known" 
representations of the group generated by /i, such as the regular representa- 
tion. In the following example, such an explicit bound is given. The crucial 
tool we use is Mackey's tensor product theorem This approach succeeds here 
because of the special features of the example and we could not use it to get 
explicit bounds in the most general case. 
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Example 35 Let n = n 3;2 be the free 2-step nilpotent Lie algebra on 3 
generators and let N = A 3i2 be the corresponding connected and simply- 
connected nilpotent Lie group. As is well-known, n is a 6-dimensional Lie 
algebra which can be realized as follows. Set V\ — V 2 — R 3 and define a Lie 
bracket on the vector space n = V\ © V 2 by 

[(X u Y,), (X 2 , Y 2 )\ = (0, 2{X 1 A X 2 )) for all X ± , X 2 , Y ± , Y 2 e R 3 , 

where X 1 AX 2 denotes the usual cross-product on R 3 . (The factor 2 appears 
here just for computational ease.) The centre of n is V 2 and the Lie group iV 
is V\ © V 2 with the product 

(x 1 ,y 1 )(x 2 ,y 2 ) = (xi + x 2 ,yi + y 2 + x x A x 2 ) for all xi,x 2 ,y 1 ,y 2 e R 3 , 

so that the exponential mapping exp : n — > N is the identity. 
Observe that, for a matrix A G GL 3 (R), we have 

A(X AY) = (det A)(A t )' 1 (X A Y) for all X,YeR 3 . 

The automorphism group Aut(iV) of N is the subgroup of GLq(R) of matrices 
qa b of the form 

(A \ 

9A ' B ~ \ B (det A) {A 1 )- 1 ) 

with A e GL 3 (R) and B e M 3 (R), so that Aut(A) is isomorphic to the 
semi-direct product GL 3 (R) x M 3 (R) for the action of GL 3 (R) by left mul- 
tiplication on the vector space M 3 (R) of 3 x 3-real matrices. 

We will identify n with n* by means of the standard scalar product 
(X,Y) H> (X\Y) on R 6 . For (x,y) and (X ,Y ) in V x © V 2 , we compute 
that Ad*(x, y)(X , Y ) = (X + x A Y , Y ). It follows that the coadjoint orbit 
of (X , 0) is {(X , 0)} and, for Y ^ 0, we have 

Ad*(A)(X , Y ) = {(X + xA Y , Y ) : x £ R 3 } 
= {(X + Y,Y ) : Ye(RY ) L } 
= {(\ Y + Y,Y ) : YE(RY ) L }. 

for Ao = (Xo|lo)/||^o|| 2 - The orbits which are not reduced to singletons are 
therefore the two-dimensional afline planes 

XoX() = {(\ Y + Y,Y ) : rG(RF ) ± }, 
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parametrized by (X ,Y ) G R x (R 3 \ {0}). 

The subgroup A = Z 3 © Z 3 is a lattice in N. The group Aut(A\iV) is 
the subgroup of Aut(iV) of automorphisms Qa,b as above given by matrices 
A G GL 3 (Z) and B G M 3 (Z). 

Fix (Ao, Y ) G Rx (R 3 \{0}). The irreducible unitary representation n\ y 
of N corresponding to the coadjoint orbit O\ xo appears in the decomposition 
of L 2 (A\N) into AMsotypical components if and only if e> Ao ,y o n(Z 3 ffiZ 3 ) ^ 0. 
This is the case if and only if Y Q G Z 3 \ {0} and Ao G ||Yo||~ 2 Ay , where 
Ay is the subgroup of Z consisting of the integers m for which tuYq G 
(RFo^+rofZ 3 . 

Let T be a subgroup of Aut(A\AT). For simplicity, we assume that V 
consists only of automorphisms g^.o with A G SL 3 (Z). We identify T with a 
subgroup of SX 3 (Z). For A G SX 3 (Z), we have 

A(O X(hY0 ) = MAt) - HYo) for fa = AoHFoHVlK^r 1 ^)!! 2 . 

The stabilizer rA 0i y of O\ y (which is the stabilizer of vrA 0i y ) in Y is therefore 

r Ao ,y = {Aer ■. A% = y }, 

and is isomorphic to a subgroup of the semi-direct product SX 2 (Z) k Z 2 . 

Let 7-L\ Xo be the isotypical component of L 2 (A\N) associated to it\ 0> y 
and U\ Xo the corresponding representation of V (see Section ; we know 
that Ux ,Y 1S equivalent to Indp A V\ ,y for a representation VA 0> y of rA Oi y 
which is strongly LP modulo its projective kernel -Pxo^o f° r some real number 
p > 1. 

The projective kernel P\ ,y of VA 0i y coincides with the subgroup of T of all 
automorphisms which fixes every point (X, Y) G O\ y ; hence, -PA ,y = {^} 
if Ao = and 

P Xo Yo = {A G T : A'Yo = Y and AY = Y for all Y G (RYo)" 1 } 
if A ± 0. 

Every ^\ y Q factorizes to a representation of a quotient of N of dimension 
3 or 4, which is isomorphic to the Heisenberg group H 3 or to the direct 
product H 3 © R. It follows that the representation VA ,y of rA 0i y is strongly 
L 6+e moc i u l p Xo Yo f or every e > (see [BeHelO] and |HoMo79j l 

Set r = rA 0i y , V = V\ 0> y , and U = U\ Q y . We claim that U m is weakly 
contained in the regular representation Ar of V on £ 2 (T). 
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Indeed, by Mackey's tensor product theorem, U m is weakly equivalent to 
the direct sum 

© ^nr-nr-nr- (V ® m ® v» ® ^) , 

71 .72 ,73 er 

where V ® V" 71 ® V 72 £g> V 73 is the tensor product of the restrictions of 
V, V 71 , V 72 and l^ 73 to T D T 7 , 1 n r 72 n r 73 . Fix 7l , 72 , 73 G r. Observe that 
T fl Tg 1 n Tq 2 Pi Tq 3 is the subgroup of elements 7 G V such that 7* fixes 
F , 7^0), 7^o) and 7^0). Set 

C/ 71i727 3 = Ind^ nr?nr , 2nr , 3 (V (9 V 71 ® V 12 ® V lz ) . 
Two cases can occur. 

• First case: There exists some i G {1, 2, 3} such that jj{Y ) is not a multiple 
of Y . Then every element T D Tq 1 fl Tq 2 fl Tq 3 fixes pointwise a plane in R 3 ; 
it follows that T fl Tq 1 n r 7 , 2 fl IT 7 , 3 is abelian and hence amenable. Therefore 
^71,7273 1S weakly contained in Ap. 

• Second case: Every 7* (Y ) is a multiple of Y , that is, every 7i belongs to the 
subgroup H = {7 G T : 7 '(Yb) G {±Yo}} • Observe that r is a subgroup of 
H of index at most 2. It can be checked that the subgroup P = P\ ,y , 
which is normal in r , is normal in H. It follows that the restriction of V li to 
r n T 71 n r 72 n r 73 is strongly L 6+£ modulo P for every i G {1, 2, 3}. Hence, 

V ®V 12 ®V lz is strongly L? modulo P and hence contained in a multiple 

Tonr 71 nr 72 nr 73 

of Indp A. Since P is amenable, it follows that U 7u72t73 is weakly 
contained in Ar- As a consequence, we see that U® 4 is weakly contained in 
Ar- 

Let fi be a probability measure on T. It follows from what we have seen 
that 

\\U n (»)\\ < ||A r ( M )ir, 

where U H is the Koopman representation of V on % = L 2 (T) ± . As a conse- 
quence, we have 

||C/°( M )|| < nia^HArC/x)!! 1 / 4 , ||C/to r (^) ||}, 

where U° and U tOT are the Koopman representations of T on Lq(A\N) and 
Lq(T). The same estimate was established in [BeHelOj Corollary 3] in the 
case where N is the Heisenberg group H%. 
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15 Proof of Theorem 
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Let if be a subgroup of Aff(A\A). The following elementary proposition 
shows that ergodicity of if on T is inherited by every subgroup of finite 
index in if. 

Proposition 36 Let H be a subgroup of AS (T) and Hi a subgroup of finite 
index in if. Assume that Lq(T) contains a non-zero Hi-invariant function. 
Then Lq(T) contains a non-zero if -invariant function. 

Proof By standard arguments involving Fourier series, there exists a uni- 
tary character \ in T \ {It} with a finite orbit under p. A [H\) and such that 
ff 2 := Hi flp^ 1 (r x ) fixes x, where r x is the stabilizer of x in Aut(T). Then 
ff 2 has finite index in H and 

seH/H 2 

is a non-zero if -invariant function in Lq(T).B 
Proof of (i) in Theorem 3] 

As is well-known, the action of a group if on a probability space (X, v) 
is weakly mixing if and only if the diagonal action of H on [X x X, v £g> v) 
is ergodic. Since T x T is the maximal factor torus of (A\N) x (A\N), we 
only have to prove the statement about ergodicity. 

So, let if be a (not necessarily countable) subgroup of Aff(A\A) acting 
ergodically on T. We have to prove that if acts ergodically on A\N. We can 
assume that iV is not abelian, otherwise there is nothing to prove. 

Set r = p a (if). Recall from Sections [T21 and [T51 that we have orthogonal 
decompositions into r x N -invariant subspaces L 2 (A\A) = L 2 (T) © % and 

i 

such that the representation Ui of T x N on H,^ is equivalent to an induced 
representation Indp^^V^, where is the stabilizer in T of some 7Tj e Sj. 
In view of the previous proposition, it suffices to prove the following 

Claim: Assume that, for some i, the subspace 7f ^ contains a non-zero if- 
invariant function. Then L q(T) contains a non-zero ffx-invariant function for 
some subgroup ifi of finite index in if. 
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To show this, set ir = 7Tj, E^ = Ej, U n = and V n = V^. Let S be a set 
of representatives for the cosets in 

T/T w * (r x N)/(T n k JV) 

with e £ 5. Then, by the definition of an induced representation, is an 
orthogonal sum 

where K carries the r„- ix iV-representation V n and where fC s = U n (s)K,. It 
follows from this that there exists a non-zero function in K which is invariant 
under H n (T^ k A/") and that r„- has finite index in V. 

Upon replacing H by the subgroup of finite index H fl (T n k iV), we can 
assume that H is contained in ix N. 

Let L n be the connected component of Ker(7r) and iV = N/L n . Observe 
that iV is not abelian, since 7r is not a unitary character of N. As seen 
in Section [TOj the action of T ff k iV on Ti n factorizes through the quotient 
nilmanifold A\N. Hence, we can assume that is trivial. 

By the proof of Proposition [311 there exists a real number p > 1 such 
that the representation V n of T n k A" is strongly L p modulo A, where A is 
the normal subgroup 

A = {(Ad(x), x- x z) :xeA,ze Z(N)}. 

We claim that H fl A has finite index in H. 

Indeed, let R = HA be the closure of HA in k N. Then the restriction 
of 14 to i? is strongly L p modulo A. 

Observe that (Ad(x), x~ x z) G A acts as multiplication with X n (z) on 
where A^ is the central character of tt. Let £ a non-zero V n (H) -invariant func- 
tion in fC. The function x H- \(V n (x)^, £)| is non-zero, belongs to L P (R/A), 
and is R invariant. It follows that R/A is a compact group. 

Let Ro be the connected component of R. Since i? is a Lie group, Rq is 
open in R. It follows that R A/A is an open (and hence closed) subgroup of 
R/A. Since R/A is compact, we conclude that RqA/ A = Rq/(R fl A) is a 
subgroup of finite index in R/A. 

On the other hand, observe that Rq C N, since R C T n tx N and since r„- 
is discrete. Observe also that 

R n A = R nz(N), 
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since Z(N) is connected (as N is simply connected). It follows that RqH A 
is a connected subgroup of the nilpotent simply connected Lie group R . But 
Ro/{Ro H A) is compact. Hence, Rq/(Ro D A) is trivial. As a consequence, 
we see that R/A is finite. This shows that H n A has finite index in if. 
Therefore, upon replacing if by if fl A, we can assume that if C A. 

The centre Z(N) being a rational subgroup of N, the subgroup A = 
AZ(N) of the nilpotent Lie group iV = N/Z(N) is a lattice. Observe that 
N is non-trivial, since N is non-abelian. The group A acts trivially on the 
factor nilmanifold A\N and hence on the associated torus T. Since T is a 
A-invariant factor torus of T, it follows that the action of if on T is not 
ergodic.B 

Proof of (ii) in Theorem [4] 

Let if be a subgroup of Aut(A\iV) with a strongly mixing action on T. 
We have to prove that the action of if on A\iV is strongly mixing. 

With the notation as in the proof of Part (i) above, the Koopman rep- 
resentation U of if on H decomposes as a direct sum U = ©JJi, where Ui 
equivalent to an induced representation lnd H Vi. It suffices to prove that, 
for every i, the matrix coefficients of U{ belong to co(if). This will follow if 
we show that the matrix coefficients of V, belong to co(if^). 

Set 7r = 7r.j and V n = Vi. Let L n be the connected component of Ker(7r) 
and A\N the corresponding if^-invariant factor nilmanifold. Since H n is 
contained in Aut(A\A), the projective kernel P of V n coincides with the 
kernel of the homomorphism ip : H n — > Aut(A\A), by Proposition [271 

We claim that P = Ker((p) is finite. Indeed, otherwise the matrix coeffi- 
cients of the Koopman representation of H n on the maximal factor torus T 
of A\N would not belong to co(H^) and this would imply that the action of 

and hence of H on T is not strongly mixing. 

Since P is finite, V n is strongly LP for some p > 1. It follows that the 
matrix coefficients of V w belong to co(H n ). This finishes the proof of Theo- 
rem m ■ 
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